Seeing small objects enlarged, and working on those objects, began with the lens. A large lens presents paired distinct views (a stereo image) to the two eyes of an operator, and a change in view (parallax) when the operator's head moves. These two depth cues are powerful for the human vision system, and enable the operator to obtain a strong three-dimensional sense of the object behind the lens. Such depth cues are important if the operator is to manipulate objects such as tools. It must be possible to position a tool correctly in the depth direction, as well as move the tool correctly across the image in the length and width directions.
Subsequently, the microscope, combining multiple lenses and other elements, brought greater enlarging power, but at the cost of poorer depth cues. Since the operator must place an eye exactly at the eyepiece, head motion needed for parallax is not possible. The standard single eyepiece microscope also provides only one view. Although weaker depth cues, such as a near object obscuring a more distant object, may still be provided to the operator, positioning a tool, precisely in the depth direction may not be possible. The binocular microscope, which provides two eyepieces but which provides the same view to both eyes, brought extra comfort for the operator because there is no need to cover or shut the second eye or ignore that eye's view. However, the human brain can still not reconstruct depth, and guiding micro-tools remains difficult.
The stereo microscope uses two separate optical paths with two objectives and two eyepieces to provide slightly different viewing angles to the left and right eyes of an operator. The stereo microscope restored binocular depth perception, but parallax remains unavailable, because the operator's head must remain stationary at the eyepieces. In addition, stereo depth perception is strong in only about half the population, so the stereo microscope brought good depth perception to only about half the users. Operators, such as micro-surgeons and those assembling microscopic electronic parts, without strong stereo depth perception, were still unable to use the stereo microscope effectively.
However, the microscope does maintain a key advantage of the simple lens: hand/eye co-location and hand/eye co-orientation. In a direct view, the visually perceived position of a hand or tool coincides almost exactly with the operator's neuromuscular sense of where it is, and in what direction its parts are oriented, through the proprioception of the operator's own joint positions. Such a match between apparent (viewed) and physical locations is referred to as hand/eye co-location. And, such a match between apparent (viewed) and physical orientations is referred to as hand/eye co-orientation. These are key requirements for guiding a tool. Enlargement necessarily sacrifices the locational match between perception and view to some extent, but the scene can still be centred on a common location. More importantly, every rotation has an axis direction and an angle of rotation, and these can match well between perception and the magnified view.
FIG. 1 shows a surgical microscope 101, according to the prior art, in which the co-orientation is exactly preserved. The microscope 101 comprises an objective lens 103 and an eyepiece 105. The operator's eye or eyes 107 (shown in profile in FIG. 1) can see vertically down through eyepiece 105, to see an enlarged view of work site 109. However, most work is more comfortable with a tilted eyepiece, so that a sitting operator can look into the eyepiece easily. FIG. 2 shows an indirect microscope 201 according to the prior art. The microscope 201 comprises a vertical object lens assembly 203 and an angled eyepiece or eyepieces 205 for the operator's eye or eyes 207 (shown in profile in FIG. 2). A system of lenses and mirrors (not shown in FIG. 2) inside the microscope delivers the view of the work site 209 from the vertical object lens assembly 203 to the tilted eyepiece or eyepieces 205. In the microscope of FIG. 2, the orientations match imperfectly, and the visual location of the work site (shown schematically at 211, although the apparent distance depends on stereo adjustments) is moved substantially from its physical site 209. Operators may learn to overcome these obstacles, but they are not trivial.
FIG. 3 shows an imaging system 301 according to WO 2008/100229, in the name of National University of Singapore, the disclosure of which is hereby incorporated by reference. In FIG. 3, the work site and its contents appear to the operator in substantially the same location and orientation as envisaged by the operator's motor cortex. Variations, for use with two collaborating operators, may be provided and will be described more fully with reference to FIGS. 4 and 5. In this specification, such a system, including the variations described below, will be referred to as a “flatscope”. A display appears on a thin flat display screen 303, and the stereo view is achieved by alternating left and right views with synchronized shutter glasses 305, worn by the operator, to control which eye sees which view, or by a means of directing different view in different directions. The subject or operand 307 of the operator's actions, shown in FIG. 3 as a pair of artery ends to be joined, appears as an enlarged operand image 309 on the display screen 303. Similarly, any tool 311 held and operated by the operator appears as an enlarged tool image 313 on the display screen 303. The close match between apparent and real locations and the precise match between apparent and real orientations assists the operator. The replacement of an optical eyepiece by a larger display screen is not unique to the system shown in FIG. 3. However, the close match between hand and eye in perceived location is missing in some recently developed systems, which place the screen away from the line between the eye and the physical operand, and typically rotate it by as much as 90°. In this specification, systems with such major deviation from hand/eye co-location are excluded from the term “flatscope”.
As shown in FIG. 3, a display screen is provided between the operator's physical work site and the operator's eyes. On the display screen, a magnified view of the work site appears to the operator in substantially the same location and orientation as it is envisaged by the operator's motor cortex. That is, there is no discrepancy between the physical and apparent locations of the work site and operand, as shown in FIG. 2. Typically the magnification is up to fifteen times, not because of technical obstacles to higher magnification, but because hand tremor makes a closer view unhelpful. If a means to reduce tremor is used, the usable magnification factor increases. With indirect means of controlling tremor, such as the use of remotely controlled robot arms at a distant location, the hand/eye co-location is specifically between where the work site is seen to be by the operator and where it is felt to be by the operator's hands, without regard to the physical location of the work site. In a preferred implementation, the display screen is substantially orthogonal to the line of view, but appropriate construction of the view permits any angle at which the screen wholly covers the eyes' view of the work site and can be clearly seen.
As already mentioned, two operators may collaborate and work on a shared magnified operand. This is particularly common among surgeons, where tasks such as suturing are easier with more than two hands. FIG. 4′ is a schematic illustration of a single-user flatscope according to the prior art, which may be used by a plurality of collaborating operators. FIG. 4 shows a single display screen 401 located between the physical work site 403 and the eyes 405 of a plurality of operators. In FIG. 4, two operators are shown, but any number of operators may collaborate. FIG. 5 is a schematic illustration of a two-user flatscope according to the prior art. FIG. 5 shows two display screens 501 located between the physical work site 503 and the eyes 505 of two operators. Analogous placements for three or four operators will be evident to persons skilled in the art.
Hand/eye co-location involves not only a sense of the direction in which the operand lies, but a sense of the distance from the eyes of each object within the work site; this is referred to as “depth”. Some sensory cues of depth are relative, such as which object obscures another and is therefore closer to the eyes. Accurate manipulation requires awareness of the absolute position in all directions. The main visual cues for absolute depth are stereo (reconstruction of distance from the differences between the views of the two eyes), and parallax (reconstruction of distance from the changes in view as the head, and thus each eye, moves). The system shown in FIG. 3 relies on an arrangement of a pair of cameras (or one camera with a split view) and optics, so that the views captured geometrically correspond to the views that would appear to eyes in the expected locations. This approach is a great improvement over previous systems but does have limitations. In particular, unless the optics move as the eyes move (which is not impossible, but creates substantial difficulty, and requires mechanical motion of parts), the approach presents two fixed views, making parallax impossible.
It is an object of the invention to provide an imaging system and method which avoids or mitigates the problems of known systems described above.